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1.  Introduction 


1,1  Project  Background 

Evaluating  Soldiers  who  are  operating  future  force  technology  in  a  live  environment  is  very 
resource  intensive.  Live  evaluations  involve  the  use  of  ranges,  vehicle  platforms  for  mounting 
the  user  interface,  numerous  test  personnel,  use  of  petroleum,  oils,  and  lubricants  (POL),  plus 
safety  and  weather  considerations.  Evaluation  of  future  force  technology  in  virtual  simulation  is 
less  resource  intensive.  It  can  be  performed  in  an  existing  battle  laboratory  setting  with  a  re-con¬ 
figurable  vehicle  with  only  changes  in  software,  and  it  can  involve  fewer  test  personnel  and  no 
POL,  weather,  or  safety  considerations.  If  the  virtual  evaluation  yields  similar  results  to  the  live 
in  terms  of  workload  and  stress,  considerable  cost  savings  could  be  achieved  in  determining 
MANPRINT  (manpower  and  personnel  integration)  requirements.  If  the  difference  is  substantial, 
then  these  data  could  be  used  in  the  development  of  algorithms  for  use  in  battle  laboratory 
simulations  and  other  models  to  emulate  live  versus  virtual  performance  degradation. 

Soldiers  in  the  future  force  (as  well  as  the  current  force)  will  have  different  types  of  missions. 

At  times.  Soldiers  may  perform  reconnaissance  or  locate  persons  of  interest,  whereas  in  other 
situations.  Soldiers  may  be  responsible  for  locating  persons  of  interest  and  protecting  against 
dismounted  personnel  who  constitute  a  threat.  Our  intent  was  to  determine  if  workload  during 
either  of  these  missions  seemed  to  excessively  tax  the  ability  of  future  Soldiers  to  perform  the 
missions. 

One  automation  tool  that  may  assist  Soldiers  with  multiple  tasks  is  autonomous  driving,  that  is, 
the  capability  of  a  vehicle  to  follow  a  pre-set  route  with  little  or  no  driver  intervention.  We  wanted 
to  determine  whether  this  technology  could  reduce  workload  and  stress,  freeing  crew  members  to 
perform  other  duties. 

Workload  and  stress  vary  by  duty  position.  Lor  example,  drivers  and  vehicle  commanders  may 
have  different  levels  of  workload  and  stress.  Mitchell,  Samms,  Glumm,  Krausman,  Brelsford,  and 
Garrett  (2004),  using  the  validated  workload  model  Improved  Performance  Research  Integration 
Tool  (IMPRINT),  found  that  the  Mounted  Combat  System  (MCS)  driver  had  the  most  instances  of 
high  workload  of  any  crew  member,  and  all  these  were  related  to  the  primary  activity  of  driving. 
Our  goal  was  to  determine  if  workload  for  either  position  exceeded  Soldier’s  abilities. 

Vehicle  speed  will  likely  affect  workload  and  stress.  Speed  often  affects  survivability  since  a 
faster  moving  vehicle  is  a  more  difficult  target  and  can  leave  danger  areas  more  quickly.  However, 
it  is  logical  to  assume  that  higher  speeds  increase  workload  and  stress  of  crew  members.  Our  intent 
was  to  determine  if  higher  speeds  represented  unacceptable  increases  in  workload  and  stress. 
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This  joint  effort  between  the  U.S.  Army  Researeh  Development  and  Engineering  Command 
(RDECOM)  and  the  U.S.  Army  Training  and  Doetrine  Command  Unit  of  Aetion  Maneuver  Battle 
Eaboratory  (UAMBL)  was  in  support  of  the  Crew  integration  and  Automation  Test  Bed  (CAT) 
Advaneed  Teehnology  Demonstration  (ATD)  program,  or  CAT-ATD.  Other  experiments  eon- 
eerning  future  eombat  vehieles  investigated  eapabilities  related  to  human-robotie  interaetions  and 
fire  eontrol  teehnologies  developed  under  the  Teehnology  for  Human-Robotie  Interaetion  in 
Soldier-Robot  Teaming  (HRI)  and  Eire  Control  Node  Engagement  (EC-NET)  Army  teehnology 
objectives  (ATOs). 

Specifically,  Experiment  1  examined  how  Soldier  decision  aids  (called  crew-aiding  behaviors 
or  CABs)  designed  to  automatically  plan  routes,  observations  posts  (OPs),  and  firing  positions 
affected  Soldier  workload  and  performance  in  terms  of  speed  and  accuracy.  In  Experiment  1, 
Soldiers  planned  a  road  march  and  tactical  movement,  with  and  without  CABs.  Experiment  1 
was  performed  in  a  stationary  virtual  simulator  described  in  section  2.  Preliminary  results 
suggested  that  the  CABs  reduced  workload  and  increased  planning  speed. 

Experiment  2  examined  how  autonomous  driving  affected  Soldier  workload  and  performance  in 
the  re-planning  of  a  route  from  a  moving  vehicle,  while  local  security  was  maintained.  Soldiers 
re-planned  a  tactical  movement  while  moving  and  maintaining  local  security  from  a  CAT  vehicle 
(described  in  section  2)  during  manual  and  autonomous  driving.  Preliminary  results  suggested  that 
autonomous  driving  decreased  workload. 

Experiment  4'  examined  how  a  target  prioritization  CAB  affected  Soldier  workload  and  perform¬ 
ance.  Soldiers  prioritized  targets  and  selected  weapons  and  ammunition  with  and  without  CABs. 
This  experiment  was  also  performed  in  a  stationary  simulator.  Results  showed  that  the  CABs 
reduced  workload  and  substantially  deceased  time  to  prioritize  targets,  select  weapons,  and  select 
ammunition. 

RDECOM-UAMBE  Experiment  06  (RUX06)  is  discussed  in  detail  by  Nunez  (2006).  These 
efforts  incorporated  UAMBE  virtual  environment  capabilities  with  live  tactical  elements  to  pro¬ 
duce  realistic  simulated  battlefield  scenarios.  Eor  example,  “live”  experiments  were  conducted 
with  actual  vehicles  and  systems  on  vehicle  ranges  9  and  10  at  Eort  Knox,  Kentucky.  “Virtual” 
elements  were  conducted  in  vehicle  mock-ups  with  computer-simulated  imagery  in  UAMBE 
building  2001  at  Eort  Knox.  Communications  and  battlefield  situational  awareness  (SA)  were 
updated  and  presented  in  near  real  time  between  the  virtual  and  live  elements. 

1.2  Research  Objective 

This  research  had  five  objectives.  The  first  was  to  compare  workload  and  stress  during  live  and 
virtual  simulation.  The  second  was  to  compare  workload  and  stress  during  varying  threat  condi¬ 
tions  (e.g.,  locating  personnel  of  interest  versus  locating  personnel  of  interest  plus  defending 
against  dismounted  threats).  However,  the  differentiation  of  these  conditions  was  compromised 


'Experiment  3  was  not  eondueted  beeause  of  time  eonstraints. 
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since  Soldiers  were  not  told  whether  threat  personnel  would  be  present  on  a  given  run.  The  third 
(in  the  live  vehiele  only)  was  to  examine  the  effeets  of  autonomous  driving  on  workload  and  stress 
The  fourth  (in  the  live  vehiele  only)  was  to  examine  the  effeets  of  duty  position  on  workload  and 
stress.  The  fifth  and  final  objeetive  (in  the  live  vehiele  only)  was  to  examine  the  effeets  of  vehiele 
speed  on  workload  and  stress. 


2.  Method 


2.1  Apparatus 

2.1.1  ACRT  Simulator 

The  Advanced  Coneepts  Researeh  Tool  (ACRT)  shown  in  figure  1  is  a  reconfigurable  simulator  and 
was  used  in  this  experiment  to  emulate  an  MCS  vehiele  in  the  Future  Combat  System  (FCS)  family 
of  systems.  Eaeh  of  the  three  ACRTs  had  three  erew  positions;  vehiele  eommander,  driver,  and 
gunner.  The  vehiele  eommander  sat  in  the  rear  position,  and  the  gunner  and  driver  sat  in  the  front. 
Eaeh  position  had  two  screens  available  to  it.  One  sereen  could  be  used  to  display  the  common 
situational  map,  while  the  other  sereen  eould  display  a  gunner’s  site,  a  driver’s  indireet  driving  view, 
or  a  sereen  showing  a  view  from  an  unmanned  ground  or  aerial  vehiele ’s  eamera.  There  were  also 
three  screens  at  the  front  of  the  vehiele  to  emulate  viewing  ports  at  the  front  of  the  vehiele.  The  two 
front  positions  had  driving  eontrols  (brake  and  gas  pedals,  steering  yoke)  and  gunner  eontrols  (a 
yoke  to  eontrol  direction  and  elevation  of  the  main  gun,  laser  switeh,  and  firing  switeh). 


Figure  1.  Reconfigurable  ACRT  simulator. 
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2,1,2  Crew  Integration  and  Automation  Test  Bed  (CAT)  Vehicle  and  Simulator 

The  CAT  Soldier-in-the-loop  (SIL)  interface  (see  figure  2)  consisted  of  three  vertically  oriented 
liquid  crystal  displays  situated  in  an  arc  in  front  of  a  seated  participant.  Each  display  was  divided 
in  two,  horizontally,  with  information  on  each  of  the  six  “screens”  provided  from  various  computer 
systems,  which  were  transparent  to  the  SIL  operation  and  the  participant.  One  type  of  information 
that  could  be  displayed  on  the  CAT  SIL  was  a  map  of  the  area  of  operation,  with  map  tools  such  as 
zoom  and  operational  graphics.  Another  type  of  information  available  was  unmanned  vehicle 
(UV)  control.  The  UV  control  information  available  included  the  types  of  UVs  available,  their 
status,  and  a  camera  view  from  one  of  the  UVs  being  controlled.  A  third  type  of  information  was 
target  acquisition  and  reports.  Soldiers  could  use  this  view  to  engage  targets  and  to  send  or  display 
reports  received  from  others.  A  drive  panel  could  also  be  selected  in  order  to  see  the  views  from 
front  and  rear  cameras  of  one’s  own  platform,  as  well  as  various  gauges  involved  in  driving  (e.g., 
speed,  compass  direction,  etc.). 


Figure  2.  CAT  SIL  (MCS)  crew  station. 


Various  types  of  information  could  be  manipulated  by  buttons  and  menus.  Lor  instance,  one  could 
display  a  driving  camera  view  and  UV  control  information  on  one  of  the  three  panels  or  a  driving 
camera  view  and  map  information  on  one  of  the  three  panels.  The  hand  controls  consisted  of 
yokes  that  could  be  used  for  steering,  and  switches  used  to  “laze”  (i.e.,  emit  coherent  light  at)  and 
fire  at  targets.  The  foot  pedals  could  be  used  for  acceleration  and  braking  of  one’s  own  platform. 
Since  the  live  vehicle  had  only  two  crew  positions  available,  there  was  one  CAT  simulator  crew 
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position  for  the  platoon  leader  in  building  2001  at  Fort  Knox.  There  were  two  CAT  crew  positions 
(driver  and  gunner)  in  a  live  vehicle  CAT  Stryker  on  a  range  at  Fort  Knox.  Combined,  these  three 
crew  positions  simulated  the  three-Soldier  crew  of  an  FCS  MCS  vehicle.  However,  both  the 
platoon  leader  and  the  other  two  crew  members  reported  that  this  positioning  of  the  platoon  leader 
in  a  separate  location  tended  to  psychologically  isolate  him  from  the  crew. 

2,2  Measures 

2.2.1  Workload 

To  measure  subjective  self-ratings  of  perceived  workload,  the  National  Aeronautics  and  Space 
Administration  (NASA)-Task  Load  Index  (TLX)  was  used.  The  NAS  A- TLX  (Hart  &  Staveland, 
1988)  is  a  multi-dimensional  rating  procedure  that  derives  an  overall  workload  score  based  on 
ratings  on  six  subscales.  The  subscales  include  Mental  Demands,  Physical  Demands,  Temporal 
Demands,  Own  Performance,  Effort,  and  Frustration.  Each  subscale  consisted  of  20  points,  for  a 
total  possible  workload  of  120.  Ratings  were  collected  on  the  questionnaire.  This  instrument  is 
shown  in  appendix  A. 

2.2.2  Stress 

One-item  rating  scales  measuring  perceived  physical  stress  level  and  mental  stress  level  each  on 
a  ten-point  scale  were  used.  The  scales  are  shown  in  appendix  A. 

2.3  Participants 

Twelve  active-duty  male  Soldiers  volunteered  for  this  experiment.  One  Soldier  was  a  Captain 
(03),  seven  Soldiers  were  Sergeants  Eirst  Class  (E7),  and  four  Soldiers  were  Staff  Sergeants  (E6). 
Military  occupational  specialties  were  primarily  Ml  Armor  Crewmen  (19K).  Nine  participants 
were  19K,  one  19D  (Cavalry  Scout),  one  14E  (Patriot  Eire  Control  Enhanced  Operator),  and  one 
25B  (Information  Systems  Operator- Analyst).  Each  vehicle,  live  and  virtual,  had  a  three-person 
crew:  driver,  gunner,  and  vehicle  commander;  personnel  were  randomly  assigned  to  each  vehicle. 
The  driver  was  responsible  for  driving  and  scanning  for  personnel  of  interest  or  threats.  The 
gunner  was  responsible  for  scanning  for  personnel  of  interest  or  threats,  engaging  threats,  and 
engaging  beyond  line  of  sight  (BEOS)  targets.  The  platoon  leader  was  responsible  for  assigning 
BLOS  targets  to  crews  and  communicating  with  crews  and  the  troop  commander  (seated  at  a 
nearby  computer).  In  the  live  vehicle,  the  vehicle  commander  (also  platoon  leader),  was  physically 
located  in  a  virtual  CAT  workstation  in  building  2001  and  communicated  with  his  crew  via  radio. 
Because  of  the  very  small  sample  size,  only  descriptive  statistics  were  used  in  this  report. 

2.4  Training 

Participants  in  the  ACRTs  received  brief  training  in  operating  the  software  to  control  the  SA 
interface  (maneuver  command  and  control  or  MC2)  and  driver  and  gunner  controls.  The  partici¬ 
pant  at  the  CAT  SIE  crew  station  (i.e.,  the  platoon  leader)  had  received  training  in  a  prior 
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experiment  in  how  to  use  the  software  to  plan  and  was  given  a  brief  refresher  about  the  system. 
Participants  in  the  CAT  Stryker  were  also  trained  in  driving  and  gunning  functions  in  a  prior 
experiment  and  were  also  given  refresher  training  and  a  safety  briefing.  Although  no  criterion- 
based  test  was  used,  experimenters  ensured  that  participants  acquired  a  baseline  level  of 
proficiency  before  the  trials  began. 


2,5  Procedures 

Participants  were  given  an  overview  of  the  experiment  and  were  familiarized  with  the  surveys. 
Baseline  measures  of  workload  and  stress  were  taken  only  in  the  virtual  condition  because  of  a 
miscommunication  in  the  field.  For  the  baseline  stress  level,  participants  were  asked  to  rate  how 
they  felt  “right  now,”  immediately  before  the  start  of  the  experimental  trials.  For  baseline  work¬ 
load,  participants  were  asked  to  rate  the  workload  experienced  in  driving  to  work  that  morning. 

We  chose  this  task  because  it  was  a  relatively  simple  (presumably)  low-workload  task  that  pro¬ 
vided  a  point  of  reference  against  which  the  reported  workload  of  other  tasks  could  be  compared. 
Participants  were  then  trained  or  given  refresher  training  in  operating  their  assigned  work  station. 
Participants  then  conducted  a  total  of  five  trials. 

Two  trials  were  conducted  on  the  first  day  of  the  experiment;  the  first  involved  identifying 
persons  of  interest  (non-combatants)  only,  and  the  second  involved  identifying  non-combatants 
and  engaging  dismounted  threat  teams  consisting  of  two  enemies  with  a  rocket-propelled  grenade 
(RPG).  On  the  second  day,  three  experimental  trials  were  conducted;  the  first  involved  non- 
combatants  and  RPG  teams,  and  the  second  involved  non-combatants  only.  The  third  involved 
non-combatants  and  RPG  teams.  After  each  experimental  trial,  participants  completed  workload, 
stress,  and  SA  surveys. 

For  the  two  crew  members  in  the  live  CAT  Stryker,  the  first  trials  on  both  days  involved  manual 
driving  (using  indirect  vision),  while  the  second  trials  involved  autonomous  driving.  The  third 
trial  on  day  2,  which  used  manual  driving,  was  at  20  miles  per  hour  (mph).  The  other  four  trials 
were  at  12  mph.  A  summary  of  trials  and  conditions  is  presented  in  table  1. 

During  all  trials,  the  MCS  platoon  maneuvered  along  a  route  to  support  an  over-watch  position 
held  by  the  reconnaissance  troop.  The  MCS  platoon  was  directed  to  engage  BLOS  targets  detected 
by  the  reconnaissance  troop  and  relayed  by  the  troop  commander  to  the  MCS  platoon  leader,  while 
maintaining  local  security  and  conducting  a  battle  damage  assessment  (BDA).  However,  since 
this  was  constant  in  all  trials,  it  was  not  considered  a  factor  in  differences  in  workload  among  the 
trials. 


Table  1.  Summary  of  trials  and  conditions. 


Trial 

Live 

Virtual 

1 

Non-combatants  only  -  Manual  driving  -  12  mph 

Non-combatants  only 

2 

Non-combatants  and  Enemy  -  Autonomous  driving  -  12  mph 

Non-combatants  and  Enemy 

3 

Non-combatants  and  Enemy-  Manual  driving  -  12  mph 

Non-combatants  and  Enemy 

4 

Non-combatants  only  -  Autonomous  driving  -  12  mph 

Non-combatants  only 

5 

Non-combatants  and  Enemy  -  Manual  driving  -  20  mph 

Non-combatants  and  Enemy 
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2,6  Analyses 


Because  of  the  small  number  of  participants  and  the  inability  to  counterbalance  conditions,  only 
descriptive  statistics  were  used  during  the  analysis.  Means  for  workload  and  stress  level  were 
compared  for  live  versus  virtual  simulation;  mission  (baseline,  non-combatants  only,  non-com¬ 
batants  and  enemy);  manual  versus  autonomous  driving  mode  (live  vehicle  only);  crew  position 
(live  vehicle  only);  and  speed  (live  vehicle  only). 


3.  Results 


3.1  Workload 

3.1.1  Virtual  Versus  Live  Simulation  and  Mission 

Because  there  was  a  slight  difference  (interaction)  between  live  and  virtual  simulation  in  how 
mission  affected  workload,  the  analyses  for  live  versus  virtual  simulation  and  type  of  mission  were 
combined.  Tables  2  and  3  show  the  data  for  workload  by  type  of  simulation  and  mission. 

Workload  in  the  virtual  simulation  (see  table  2)  was  very  light,  only  reaching  about  16%  (19.21  of 
a  possible  120)  of  total  possible  workload.  Conversely,  workload  in  the  live  simulation  mission  (see 
table  3)  was  considerably  higher,  reaching  about  46%  (55.50)  of  total  possible  workload.  Figure  3 
provides  a  comparison  between  virtual  and  live  simulation  across  missions. 


Table  2.  Workload  in  virtual  simulation  by  mission. 


Mission 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frustration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Baseline  (n=16) 

2.25 

2.25 

2.00 

6.94 

3.88 

2.06 

19.38 

Non-combatants 
only  (n=14) 

2.29 

1.86 

2.07 

7.14 

1.79 

4.07 

19.21 

Non-combatants 
and  Enemy 
(n=19) 

1.79 

1.63 

2.05 

6.37 

1.68 

2.74 

16.25 

Table  3.  Workload  in  live  simulation  by  mission. 


Mission 

Mental 

Workload 

Physical 

Workload 

Temporal 

Workload 

Performance 

Workload 

Effort 

Workload 

Frustration 

Workload 

Total 

Workload 

Baseline 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Non-combatants 
only  (n=4) 

7.50 

4.50 

11.50 

15.00 

13.00 

4.00 

55.50 

Non-combatants 
and  Enemy 
(n=4) 

9.50 

5.75 

7.75 

15.25 

12.00 

5.25 

55.50 

There  were  relatively  minor  differences  between  missions  during  virtual  and  live  simulation.  In 
the  virtual  simulation,  overall  workload  for  the  higher  threat  condition  (where  RPG  teams  were 
present)  was  actually  slightly  lower  than  for  any  other  condition,  including  baseline.  However, 
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since  participants  were  not  aware  of  whether  threats  were  present  until  and  unless  they  encountered 
them,  the  tasks  in  both  threat  conditions  (i.e.,  scan  for  threats  versus  persons  of  interest)  were  quite 
similar.  The  subscales  of  mental,  physical,  performance,  and  effort  workloads  were  lower  for  the 
higher  threat  condition  than  any  for  other  condition.  In  the  live  condition,  overall  workload  was 
the  same  during  lower  and  higher  threat  conditions.  Mental,  physical,  and  frustration  workloads 
were  at  least  a  point  higher  in  the  higher  threat  condition,  while  temporal  and  effort  workloads  at 
least  a  point  higher  in  the  low  threat  condition.  Figure  4  provides  a  comparison  between  each 
mission  type  across  each  of  the  six  subscales  of  workload. 
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Figure  3.  Workload  by  mission  type. 
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Figure  4.  Workload  subscales  by  mission  type. 
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3.1,2  Crew  Position  (live  simulation  only) 


Data  for  crew  position  were  colleeted  for  the  live  vehiele  only.  Table  4  shows  data  for  workload 
by  erew  position.  Overall  workload  for  the  gunner  was  higher  than  for  the  driver.  This  comes 
mainly  from  the  larger  workload  levels  for  the  performanee  and  (especially)  effort  subscales. 


Table  4.  Workload  by  crew  position. 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Driver 

(n-4) 

8.00 

5.25 

9.75 

13.25 

6.00 

5.00 

47.25 

Gunner 

(n=4) 

9.00 

5.00 

9.50 

17.00 

19.00 

4.25 

63.75 

One  might  expect  that  workload  of  gunners  and  drivers  would  be  affected  by  mission.  That  is, 
when  only  non-combatants  were  present,  the  driver  had  to  drive  and  (to  the  extent  possible) 
identify  personnel,  whereas  the  gunner  had  to  identify  personnel  and  engage  BLOS  targets. 
However,  in  the  trials  when  enemy  were  present,  the  gunner  had  to  scan  for  personnel,  engage 
threats,  and  engage  BLOS  targets,  thus  increasing  workload.  Tables  5  and  6  examine  workload  by 
condition  and  crew  position.  The  hypothesis  is  not  quite  confirmed,  in  that  in  the  non-eombatants- 
only  trials  (see  table  5),  gunners  had  a  slightly  higher  overall  workload  than  drivers  (beeause  of 
differences  in  performance  and  effort  workload),  although  drivers’  mental,  physical,  temporal,  and 
frustration  workloads  were  higher.  However,  in  the  trials  where  non-combatants  and  enemy  were 
present  (see  table  6),  gunners  had  a  far  higher  overall  workload  than  drivers,  thus  showing  the 
added  workload  when  gunners  were  scanning  for  personnel  and  engaging  threats. 


Table  5.  Workload  by  crew  position  by  mission  (non-combatants  only). 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Driver 

(n=2) 

10.00 

6.00 

13.00 

11.50 

6.00 

6.50 

53.00 

Gunner 

(n=2) 

5.00 

3.00 

10.00 

18.50 

20.00 

1.50 

58.00 

Table  6.  Workload  by  crew  position  by  mission  (non-combatants  and  enemy). 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Driver 

(n=2) 

6.00 

4.50 

6.50 

15.00 

6.00 

3.50 

41.50 

Gunner 

(n=2) 

13.00 

7.00 

9.00 

15.50 

18.00 

7.00 

69.50 

3,1.3  Autonomous  Versus  Manual  Driving  Mode  (live  simulation  only) 

Since  the  high-speed  (20-mph)  trial  was  conducted  only  in  the  manual  driving  mode,  comparison 
between  driving  modes  was  only  condueted  in  the  low-speed  (12-mph)  trials.  Table  7  presents 
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data  for  workload  by  driving  mode.  There  is  very  little  differenee  in  overall  workload  between 
autonomous  and  manual  driving.  Temporal  workload  is  aetually  more  than  five  points  higher  in 
the  autonomous  mode. 


Table  7.  Workload  by  driving  mode. 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Autonomous 

(n-4) 

7.50 

4.50 

11.50 

15.00 

13.00 

4.00 

55.00 

Manual 

(n=2) 

8.50 

4.50 

6.00 

15.50 

13.00 

4.00 

53.50 

However,  workload  by  driving  mode  should  be  affeeted  by  crew  position.  That  is,  driver  workload 
should  be  reduced  more  in  the  autonomous  mode  than  should  gunner  workload.  Tables  8  and  9 
examine  workload  by  driving  mode  and  crew  position.  Surprisingly,  driver  workload  actually 
increases  in  the  autonomous  mode,  especially  for  mental  and  temporal  workload.  One  possible 
reason  for  this  finding  is  that  it  is  difficult  to  control  the  speed  of  the  CAT  in  the  auto-pilot  mode. 
The  mission  required  instances  when  the  driver  wanted  to  “creep,”  which  was  much  easier  in  the 
manual  mode  than  in  auto-pilot.  Gunner  workload  decreases  in  the  autonomous  mode,  especially 
for  mental  workload.  Perhaps  in  the  autonomous  mode,  the  gunner  offloads  some  workload  onto 
the  driver,  since  the  driver  should  have  fewer  driving  responsibilities.  Perhaps  the  driver  is 
expected  to  assume  more  scanning  responsibilities  during  autonomous  driving.  Admittedly,  this  is 
a  speculation  derived  from  a  very  small  number  of  observations.  Figure  5  provides  a  comparison 
of  workload  by  driving  mode  across  crew  position. 


Table  8.  Workload  by  driving  mode  by  crew  position:  driver. 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Autonomous 

(n-2) 

10.00 

6.00 

13.00 

11.50 

6.00 

6.50 

53.00 

Manual 

(n=l) 

7.00 

5.00 

8.00 

15.00 

6.00 

4.00 

45.00 

Table  9.  Workload  by  driving  mode  by  crew  position:  gunner. 


Position 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Workload 

Autonomous 

(n-2) 

5.00 

3.00 

10.00 

18.50 

20.00 

1.50 

58.00 

Manual 

(n=l) 

14.00 

4.00 

4.00 

16.00 

20.00 

4.00 

62.00 

10 


Workload  by  Driving  Mode 


(A 

®o 
■E  CM 

D£  o 

TS  (O 

o  §, 

m 

O  S 


Driver 


Gunner 


Crew  Position 


Figure  5.  Workload  by  driving  mode. 


3,1,4  Driving  Speed  (live  simulation  only) 

Overall  workload  was  slightly  higher  in  the  higher  speed  condition  in  the  manual  driving  mode. 
This  was  mainly  because  of  higher  ratings  on  the  physical,  temporal,  frustration  scales.  This 
makes  sense  since  physical  actions,  such  as  steering,  would  have  to  be  done  faster  (temporal) 
during  higher  speed  and  would  thus  increase  those  aspects  of  workload.  If  actions  seemed  to 
have  to  taken  place  too  quickly,  that  could  lead  to  more  frustration. 

As  shown  in  table  10,  autonomous  driving  did  not  reduce  workload;  in  fact,  workload  was  slightly 
higher  during  autonomous  driving.  This  resulted  from  higher  temporal  workload  in  the  autono¬ 
mous  condition,  suggesting  that  things  seemed  to  happen  faster  in  the  autonomous  mode.  In  addi¬ 
tion,  the  increased  workload  may  be  attributable  to  the  driver  acting  as  systems  monitor,  which 
tends  to  increase  cognitive  workload  primarily  because  of  attention  demands. 


Table  10.  Driving  speed  by  driving  mode. 


Speed  and  Driving 
Mode 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frnstration 

Total 

12  mph  &  Autonomous 
(n=4) 

7.50 

4.50 

11.50 

15.00 

13.00 

4.00 

55.50 

12  mph  and  Manual 
(n=2) 

10.50 

4.50 

6.00 

15.50 

13.00 

4.00 

53.50 

20  mph  and  Manual 
(n=2) 

8.50 

7.00 

9.50 

15.00 

11.00 

6.50 

57.50 
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3,2  Stress 


3.2,1  Virtual  Versus  Live  Simulation  and  Mission 

Tables  1 1  and  12  present  stress  data  for  the  two  types  of  simulations  and  missions.  Stress  was 
higher  for  live  simulation  during  both  types  of  missions,  although  still  rather  low  on  a  ten-point 
scale.  There  was  very  little  difference  in  stress  between  the  two  mission  types.  There  was  no 
difference  between  mission  types  in  live  simulation  and  a  small  difference  suggesting  that  the 
higher  threat  mission  was  slightly  less  stressful  than  the  non-combatant  condition  in  virtual 
simulation.  This  is  similar  to  the  workload  data. 


Table  1 1 .  Stress  in  virtual  simulation  by  mission. 


Mission 

Physical  Stress  Level 

Mental  Stress  Level 

Baseline  (n=16) 

1.19 

1.31 

Non-combatants  only  (n=14) 

1.21 

1.36 

Non  combatants  and  enemy  (n=19) 

1.16 

1.32 

Table  12.  Stress  in  live  simulation  by  mission. 


Mission 

Physical  Stress  Level 

Mental  Stress  Level 

Baseline  (NA) 

NA 

NA 

Non-combatants  only  (n=4) 

2.75 

3.25 

Non  combatants  and  enemy  (n=4) 

2.75 

3.24 

3,2.2  Crew  Position  (live  simulation  only) 

Table  13  shows  data  for  stress  by  crew  position.  Stress  was  slightly  greater  for  the  gunner  than 
for  the  driver. 


Tablets.  Stress  by  crew  position. 


Position 

Physical  Stress  Level 

Mental  Stress  Level 

Driver  (n=4) 

2.50 

3.25 

Gunner  (n=4) 

3.00 

3.25 

As  with  workload,  tables  14  and  15  show  stress  by  crew  position  by  mission.  Also  as  with  work¬ 
load,  in  the  trials  when  only  non-combatants  were  present,  stress  was  similar  between  positions, 
with  higher  physical  stress  for  gunners  and  mental  stress  for  drivers.  However,  when  the  enemy 
were  present  as  well,  both  physical  and  mental  stress  were  higher  for  gunners. 


Table  14.  Stress  by  crew  position  by  mission  (non-combatants  only). 


Position 

Physical  Stress  Level 

Mental  Stress  Level 

Driver  (n=2) 

2.50 

3.50 

Guimer  (n=2) 

3.00 

3.00 
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Table  15.  Stress  by  crew  position  by  mission  (non-combatants  and  enemy). 


Position 

Physical  Stress  Level 

Mental  Stress  Level 

Driver  (n=2) 

2.50 

3.00 

Gunner  (n=2) 

3.00 

3.50 

3,2.3  Manual  Versus  Autonomous  Driving  (live  simulation  only) 

Since  the  high-speed  (20-mph)  trial  was  run  in  the  manual  driving  mode  only,  this  eomparison 
involved  only  the  low-speed  (12-mph)  trials.  Table  16  presents  data  for  stress  by  driving  mode. 
Stress  was  slightly  greater  in  the  autonomous  mode.  Tables  17  and  18  and  figure  6  show  stress 
by  driving  mode  for  eaeh  erew  position.  Physieal  stress  was  slightly  higher  for  the  driver  in  the 
manual  mode,  but  mental  stress  was  slightly  higher  in  the  autonomous  mode,  as  was  workload. 
For  the  gunner,  both  types  of  stress  were  higher  in  the  manual  mode,  which  suggests  an  off¬ 
loading  of  responsibility  to  the  driver  in  autonomous  driving  mode. 


Table  16.  Stress  by  autonomous  versus  manual  driving. 


Driving 

Physical  Stress  Level 

Mental  Stress  Level 

Autonomous  (n=4) 

2.75 

3.25 

Manual  (n=2) 

2.00 

2.50 

Table  17.  Stress  by  autonomous  versus  manual  driving:  driver. 


Driving 

Physical  Stress  Level 

Mental  Stress  Level 

Autonomous  (n=2) 

2.50 

3.50 

Manual  (n=l) 

3.00 

3.00 

Table  18.  Stress  by  autonomous  versus  manual  driving:  gunner. 


Driving 

Physical  Stress  Level 

Mental  Stress  Level 

Autonomous  (n=2) 

3.00 

3.00 

Manual  (n=l) 

1.00 

2.00 

3,2,4  Driving  Speed  (live  simulation  only) 

Sinee  high  speed  driving  was  done  only  in  the  manual  mode,  the  following  data  are  for  manual 
mode  only.  Table  19  shows  data  for  stress  by  driving  speed  and  figure  7  depiets  the  relationship 
between  stress  and  speed.  As  expeeted,  stress  was  higher  at  higher  speeds,  with  mental  stress 
being  higher  than  physieal  stress  in  both  speed  eonditions. 
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Figure  6.  Stress  by  driving  mode  and  erew  position. 


Table  19.  Stress  by  driving  speed. 


Speed 

Physical  Stress  Level 

Mental  Stress  Level 

12  mph  (n=2) 

2.00 

2.50 

20  mph  (n=2) 

3.50 

4.00 
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Figure  7.  Stress  by  speed. 
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4.  Discussion 


Concerning  the  first  objeetive,  eomparing  live  and  virtual  workload  and  stress,  both  workload 
and  stress  were  higher  in  the  live  versus  the  virtual  environment.  This  makes  sense  sinee  the  live 
vehiele  may  be  liable  to  mishaps  and  bears  eloser  attention,  whereas  a  virtual  one  does  not. 

Thus,  the  virtual  simulation  may  not  have  provided  the  same  realism  level  of  vehiele  mishap  as 
the  live  simulation.  This  finding  is  also  supported  by  the  analyst’s  observations  that  vehicles 
moved  faster  in  the  virtual  simulation  than  in  the  live  environment.  Thus,  differences  between 
workload  and  stress  in  a  live  versus  virtual  mode  appear  to  be  speeifie  to  the  task  performed.  In 
a  task  involving  driving,  virtual  simulation  does  not  appear  to  provide  a  realistic  assessment  of 
workload  or  stress. 

Coneerning  the  seeond  objective,  involving  workload  and  stress  by  type  of  mission,  there  were 
very  few  differenees  in  workload  and  stress  as  a  funetion  of  the  type  of  mission.  Apparently, 
looking  for  personnel,  whether  merely  eivilians  (non-threats)  or  mixed  threat  and  non-threat,  is 
roughly  equivalent  in  workload,  espeeially  when  one  does  not  know  whether  threats  are  present. 
The  types  of  tasks  (seanning  the  environment  and  driving)  are  similar  in  either  mission,  sinee  one 
must  be  alert  for  potential  threats  in  either  case.  In  the  live  simulation,  mental,  physical,  and 
frustration  workloads  were  slightly  higher  in  the  eondition  where  there  were  enemy  targets, 
whieh  suggests  that  identifying  and  servieing  threats  required  slightly  more  thinking  (to  eonfirm 
a  threat),  action  (to  engage  enemy  personnel),  and  frustration  (perhaps  in  uneertainty  that  an 
object  engaged  was  a  threat.  However,  it  does  not  appear  that  workload  or  stress  for  either  of  the 
two  missions  severely  taxed  Soldier  capabilities. 

Coneerning  the  third  objeetive,  examining  workload  and  stress  by  driving  mode,  it  appears  that 
although  workload  was  about  the  same  in  both  driving  modes,  partieipants  found  the  autonomous 
mode  more  stressful.  This  higher  stress  suggests  that  eontrolling  vehiele  speed  may  have  been 
more  diffieult  in  the  autonomous  mode.  The  autonomous  mobility  mode  had  only  a  single  speed 
setting,  and  the  vehiele  aecelerated  to  that  speed  and  tried  to  maintain  it.  A  large  part  of  the 
experiment  involved  eresting  hills  to  sean  the  area  ahead  of  the  vehiele.  In  the  autonomous 
mode,  the  driver  had  to  keep  stopping  and  starting  the  autonomous  mode  in  order  to  simulate 
“creeping”  forward.  In  faet,  temporal  workload  was  higher  in  autonomous  mode,  suggesting  that 
partieipants  spent  more  time  monitoring  vehiele  movement.  Further,  sinee  gunner  workload 
deereased  in  the  autonomous  mode,  perhaps  the  gunner  offloaded  responsibilities  to  the  driver. 
The  driver,  however,  perhaps  beeause  of  an  inability  to  easily  eontrol  speed,  felt  the  need  to 
monitor  vehiele  movement  and  assume  additional  seanning  responsibilities  as  well.  It  seems  that 
the  potential  benefit  of  automation  to  reduee  workload  and  free  operators  for  other  tasks  does  not 
appear  to  be  realized  in  this  partieular  researeh.  These  results  are  different  from  Experiment  2  in 
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RUX06,  where  autonomous  driving  redueed  workload.  One  possible  explanation  for  these 
different  results  posited  by  Nunez  (2006)  was  that  Experiment  6  involved  re-planning  in  a  seeure 
environment  (i.e.,  with  no  enemy  threat).  Therefore,  it  would  be  less  risky  to  pay  more  attention 
to  re-planning  and  less  to  what  was  happening  outside  the  vehiele  in  Experiment  2.  In  the 
eurrent  experiment,  with  an  enemy  threat.  Soldiers  wanted  to  deeelerate  the  vehiele  at  the  erest 
of  a  hill  or  when  rounding  a  bend.  Autonomous  driving  only  allowed  a  eonstant  speed  and  a 
eourse  to  be  set.  The  vehiele  had  to  be  started  and  stopped  to  adjust  speed  in  route,  thus  perhaps 
adding  workload  not  required  in  Experiment  2. 

Coneerning  the  fourth  objeetive,  workload  and  stress  by  position,  workload  and  stress  were 
slightly  higher  for  gunners  overall,  but  this  was  greatly  affeeted  by  mission  type.  That  is,  in  the 
trials  when  enemy  were  present,  the  workload  (and  to  a  lesser  extent  stress)  for  gunners  was 
mueh  higher  than  for  drivers  (and  also  mueh  higher  than  gunners’  workload  in  trials  when 
eivilians  only  were  present).  The  inereased  workload  for  gunners  in  the  trials  when  enemy  are 
present  is  likely  attributable  to  their  having  to  simultaneously  sean  for  personnel,  distinguish 
eivilians  from  enemy,  and  engage  the  enemy  personnel.  This  finding  was  different  from  that  of 
Mitehell  et  al.  (2004)  but  again  eould  be  attributable  to  mission  differenees.  In  the  IMPRINT 
model,  the  assumption  was  that  the  driver  was  driving  eonstantly,  whereas  the  gunner  only  had 
to  oeeasionally  perform  his  task.  In  the  mission  when  an  enemy  threat  was  present,  the  gunner 
had  to  eonstantly  sean  for  eivilian  personnel  and  enemy  and  engage  the  enemy  when  present. 
However,  even  in  the  more  demanding  mission,  workload  for  the  gunner  was  only  about  58%  of 
the  maximum  possible  workload,  and  thus  it  appears  within  the  eapabilities  of  the  Soldier  to 
perform. 

Higher  driving  speed  also  resulted  in  higher  workload  and  stress  than  lower  speeds.  Tasks, 
whether  driving,  seanning,  or  engaging,  were  performed  faster  at  higher  vehiele  speed.  This  was 
manifested  partieularly  in  physieal  workload  (harder  physieal  work)  and  temporal  workload  (less 
time  to  do  tasks).  However,  even  at  the  higher  driving  speed,  workload  was  less  than  half  the 
maximum  possible  workload  and  was  thus  likely  well  within  the  eapabilities  of  the  Soldier. 

These  results,  although  based  on  a  small  sample  size,  suggest  that  workload  and  stress  are  very 
task  speeifie.  When  the  task  is  operating  a  moving  vehiele  and  seouting  for  personnel  (eivilians, 
enemy,  or  both),  workload  and  stress  were  higher  in  a  live  environment.  Also,  autonomous 
driving  did  not  reduee  workload,  again,  perhaps  beeause  of  an  inability  to  adjust  speed  in  this 
mode  and  offloading  of  work  by  the  gunner  in  the  autonomous  driving  eondition.  Also,  eontrary 
to  previous  researeh,  gunners  had  higher  workloads  than  drivers  in  this  study,  partieularly  when 
enemy  targets  were  present,  perhaps  beeause  gunner  tasks  were  eonstant  in  this  study. 

This  researeh  suggests  that  examination  of  deeision  aids  or  CABs  sueh  as  autonomous  driving 
should  be  performed  during  different  eonditions,  sueh  as  live  and  virtual  simulation.  A  deeision 
or  performanee  aid  may  prove  effeetive  in  some  eonditions  but  not  others,  and  this  information 
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may  be  useful  in  judging  the  effectiveness  of  an  aid  or  under  what  conditions  it  should  be 
employed. 
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Appendix  A.  Measures 


NASA  TLX  Workload  Assessment  Instruetions 

We  are  interested  in  the  “workload”  you  experieneed  during  this  seenario.  Workload  is  something 
experieneed  individually  by  eaeh  person.  One  way  to  find  out  about  workload  is  to  ask  people  to 
deseribe  what  they  experieneed.  Workload  may  be  eaused  by  many  different  faetors  and  we 
would  like  you  to  evaluate  them  individually.  The  set  of  six  workload  rating  faetors  was 
developed  for  you  to  use  in  evaluating  your  experienees  during  different  tasks.  Please  read  them. 
If  you  have  a  question  about  any  of  the  seales  in  the  table,  please  ask  about  it.  It  is  extremely 
important  that  they  be  elear  to  you. 


Definitions 


Title 

End  Points 

Descriptions 

MENTAL 

DEMAND 

Low  /  High 

How  much  mental  and  perceptual  activity  was  required  (that  is,  thinking, 
deciding,  calculating,  remembering,  looking,  searching,  etc.)?  Was  the  task 
easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

PHYSICAL 

DEMAND 

Low  /  High 

How  much  physical  activity  was  required  (that  is,  pushing,  pulling,  turning, 
controlling,  activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk, 
slack  or  strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low  /  High 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the 
tasks  or  task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid 
and  frantic? 

PERFORMANCE 

Poor / Good 

How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the 
task?  How  satisfied  were  you  with  your  performance  in  accomplishing  these 
goals? 

EFFORT 

Low  /  High 

How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your 
level  of  performance? 

FRUSTRATION 

LEVEL 

Low  /  High 

How  insecure,  discouraged,  irritated,  stressed,  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent  did  you  feel  during  the  task? 

We  want  you  to  evaluate  workload  in  two  different  ways.  First,  rate  the  workload  on  eaeh  faetor 
on  a  seale.  Eaeh  seale  has  two  end  deseriptions,  and  20  slots  (hash  marks)  between  the  end 
deseriptions.  Plaee  an  “x”  in  the  slot  (between  the  hash  marks)  that  you  feel  most  aeeurately 
refleets  your  workload. 

Next,  we  want  you  to  eompare  the  various  workload  faetors.  This  eomparison  is  a  teehnique 
developed  by  NASA  to  evaluate  the  relative  importanee  of  the  six  workload  sourees  you  used  to 
rate  the  workload  you  experieneed.  The  proeedure  is  simple:  you  have  a  sheet  with  a  series  of 
pairs  of  workload  sourees  (for  example.  Effort  vs.  Mental  Demands).  We  want  you  to  ehoose 
whieh  of  the  sourees  was  more  important  to  your  experienee  of  the  workload  in  the  task  that  you 
performed.  You  will  fill  out  a  separate  sheet  for  eaeh  task. 
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For  each  pair  of  sources,  circle  the  source  that  is  the  more  important  contributor  to  the  workload 
for  the  specific  task  you  are  rating.  For  example,  for  the  first  comparison,  if  effort  contributes  to 
workload  more  than  performance,  circle  effort. 

After  you  have  finished  the  entire  series,  we  will  be  able  to  use  the  pattern  of  your  choices  to 
create  a  weighted  combination  of  ratings  into  a  summary  workload  score. 

We  ask  you  to  evaluate  your  workload  for  this  scenario.  This  includes  all  the  duties  involved  in 
your  job  (e.g.,  preparing  your  workstation,  using  displays  and  controls  at  your  workstation). 


Participant  ID; _  Date: _  Time; _  Experiment; _ Condition; _ 

TLX  Workload  Scale 

Please  rate  your  workload  by  putting  a  mark  on  each  of  the  six  scales  at  the  point  which  matches 
your  experience. 

Mental  Demand  _ | _ | _ | _ | _ | _ | _ | _ | _ 

Very  Low  Very  High 


Physical  Demand  — ' — — ' — — ' — — ' — — ' — — ' — — ' — — ' — — ' — — ' — 

Very  Low  Very  High 


Temporal  Demand  — ' — — I — — I — — I — — I — — I — — I — — I — — I — — I — 

Very  Low  Very  High 


Performance 


Very  Poor 


Very  Good 


Effort  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Very  Low  Very  High 


Frustration 


Very  Low 


Very  High 
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Participant  ID; _  Date: _  Time: _  Experiment: _ Condition: _ 

Subjective  Stress  Rating  Scale 

1.  The  seale  below  represents  a  range  of  how  PHYSICALLY  stressful  the  mission  might  be. 
Cheek  the  bloek  indieating  how  PHYSICALLY  stressful  the  mission  you  just  partieipated  in 
was. 


Task 

Not  at 
All 

Stressful 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Most 

Possible 

Stress 

10 

a.  Overall  stress 

2.  The  seale  below  represents  a  range  of  how  MENTALLY  stressful  the  mission  might  be. 
Cheek  the  bloek  indieating  how  MENTALLY  stressful  the  mission  that  you  just  participated  in 
was. 


Task 

Not  at 
All 

Stressful 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Most 

Possible 

Stress 

10 

a.  Overall  stress 
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